[Kernel][SM70] Default 8 KiB TP4 push all-reduce - #299
Merged
yangzhuxinyzx merged 9 commits intoAug 25, 2026
Merged
Conversation
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Adds opt-in selector-alignment capture and analysis without changing the default inference hot path.\n\nSigned-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Enable the validated capability- and shape-bounded SM70 long-prefill GQA route by default, with exact fallback on typed workspace OOM.\n\nSigned-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Add capability-based prompt lookup while applying proposals only when the full DFlash2 query can be skipped.\n\nSigned-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Expose explicit per-request OCR resolution and bounded crop controls without any model/checkpoint identity defaults. Keep the existing gundam behavior as the default and use one processor instance for producer/counting consistency.\n\nTests: 24 focused processor tests passed; changed-file pre-commit passed. The all-files CI failure modified 227 historical baseline files with zero overlap with this PR.\n\nSigned-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Default the measured SM70 QPN8 and metadata-free PP wins only under exact device/topology/concurrency/operator/tensor contracts, with explicit rollback and TurboMind fallback. Add matched aggregate-quality tooling that reports directional sample flips without requiring greedy identity.\n\nPerformance: QPN8 +8.63%; static PP adds 0.22%. Paired GSM8K remains 63/64 with zero invalid answers; operator outputs are finite with rel-L2 <= 6.05e-4 and max-abs <= 0.00390625.\n\nTests: 47 focused tests and changed-file pre-commit passed. The all-files CI failure modified 226 historical baseline files with zero overlap with this PR.\n\nSigned-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
yangzhuxinyzx
changed the title
yangzhuxinyzx
marked this pull request as ready for review
August 25, 2026 19:08
yangzhuxinyzx
deleted the
agent/v100-dsv4-pp2tp4-exact-hotpath-20260825-172438
branch
August 26, 2026 07:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Extend the existing two-epoch SM70 TP4 push collective to the exact 8-KiB FP16 decode payload and default it under the validated inference-engine contract.
Admission depends only on SM70 capability, fully connected TP4, active CUDA Graph capture, FP16 dtype, and an exact 8-KiB or existing 80-KiB payload. Other devices, topologies, dtypes, sizes, and eager calls retain the pull collective. Runtime selection does not inspect model, checkpoint,
model_type, or architecture identity.VLLM_SM70_TP4_PUSH_ALLREDUCE=0is the rollback.Four CTAs cover all 512 packed elements of the 8-KiB payload. The existing allocation is sized for the larger 80-KiB route, so the new dispatch adds no buffer growth.
Performance and numerical result
+3.57%), 16.903 -> 16.321 ms/token (-0.583 ms/token).Independent unchanged control processes produce different internally stable greedy streams because of pre-existing cross-process TurboMind autotuning. Since the collective is isolated by a bitwise operator oracle, those token hashes are diagnostic and not a greedy-identity quality gate.
Test Plan
Test Result
git diff --check: passed.